Skip to content

feat(models): effectify ModelsDev as Service#25434

Merged
kitlangton merged 6 commits intodevfrom
kit/models-dev-service
May 2, 2026
Merged

feat(models): effectify ModelsDev as Service#25434
kitlangton merged 6 commits intodevfrom
kit/models-dev-service

Conversation

@kitlangton
Copy link
Copy Markdown
Contributor

Summary

Stacks on #25429. Replaces ModelsDev's Promise-based, module-state-heavy implementation with an Effect-native Service that uses `AppFileSystem` and `HttpClient` instead of raw `fetch` + `Filesystem.*` helpers. The periodic refresh moves from a module-load `setInterval` to a scoped fork.

What's now Effect-native

  • File IO: `AppFileSystem.Service` — `fs.stat`, `fs.readJson`, `fs.writeWithDirs` (was `Filesystem.stat/readJson/write` Promise helpers)
  • HTTP: `HttpClient` from `effect/unstable/http` with `HttpClientRequest.get` + `filterStatusOk` + `withTransientReadRetry` (was raw `fetch` + `AbortSignal.timeout`)
  • Cache: closure state in the Service, populated lazily on first `get()`
  • Periodic refresh: `Effect.forkScoped(refresh().pipe(Effect.repeat(Schedule.fixed("60 minutes"))))` — auto-cleanup with the Service's scope (was top-level `setInterval(...).unref()`)

What stays Promise-wrapped

  • `Flock.withLock` — cross-process file lock; in-process Effect semaphores can't coordinate concurrent opencode CLIs writing the same cache file
  • `import('./models-snapshot.js')` — one-time dynamic import for the bundled snapshot fallback

Effect-context callers converted

File Before After
`provider/provider.ts` `yield* Effect.promise(() => ModelsDev.get())` `yield* modelsDevSvc.get()` (yielded at layer top)
`server/.../httpapi/handlers/provider.ts` same `yield* ModelsDev.Service.use(s => s.get())`
`server/routes/instance/provider.ts` same same
`cli/cmd/models.ts` `yield* Effect.promise(() => ModelsDev.refresh(true))` `yield* ModelsDev.Service.use(s => s.refresh(true))`

Promise-context legacy callers

`cli/cmd/github.ts` and `cli/cmd/providers.ts` (still vanilla `async (args) => ...` yargs handlers, not yet on `effectCmd`) keep calling `ModelsDev.get()` / `ModelsDev.refresh()`. Those are now thin `AppRuntime.runPromise(Service.use(...))` wrappers using the shared memoMap — so the Service instance and cache are the same one Effect callers see.

Followup notes

  • Once `github.ts` and `providers.ts` migrate to `effectCmd`, the Promise compat shim at the bottom of `models.ts` can drop.
  • `Flock` itself could become a Service that wraps the cross-process lock — separate concern.

Tests

  • `bun run dev models` / `bun run dev models nonexistent` — both work
  • `bun run test test/provider/provider.test.ts test/server/httpapi-provider.test.ts` — 78/78
  • `bun run typecheck` — clean

kitlangton added 2 commits May 2, 2026 12:34
Replaces the Promise-based, module-state-heavy ModelsDev with an
Effect-native Service that uses AppFileSystem (fs.stat / fs.readJson /
fs.writeWithDirs) and HttpClient (HttpClientRequest + filterStatusOk +
withTransientReadRetry) instead of raw fetch + Filesystem.* helpers.
Periodic refresh moves from a module-load setInterval to a scoped fork
that gets cleaned up with the Service's scope.

Effect-context callers yield ModelsDev.Service:
- src/provider/provider.ts (yields modelsDevSvc once at layer top)
- src/server/routes/instance/httpapi/handlers/provider.ts
- src/server/routes/instance/provider.ts
- src/cli/cmd/models.ts (drops the Effect.promise wrap)

Promise-context legacy callers (cli/cmd/github.ts, cli/cmd/providers.ts)
keep using ModelsDev.get() / ModelsDev.refresh(), which now route
through a tiny ManagedRuntime + Service wrapper that shares memoMap with
AppRuntime — so Effect callers and Promise callers see the same Service
instance and cache.

What stays Promise-wrapped:
- Flock.withLock for cross-process file coordination (in-process Effect
  semaphores can't coordinate concurrent CLIs writing the same cache)
- The dynamic import('./models-snapshot.js') one-time bootstrap

Smoke tested: bun run dev models, bun run dev models nonexistent.
Provider + httpapi-provider tests pass (78/78). Typecheck clean.
Three review findings + a real bug:

1. Replace 'Effect.promise(() => Flock.withLock(... runPromise(fetchApi) ...))'
   with 'Effect.scoped(Effect.gen { Flock.effect(lockKey); fetchAndWrite })'.
   Flock.effect is the existing scoped acquire/release; the runPromise
   re-entry inside the Promise lock body was dropping the parent fiber's
   tracing span, scope, and interruption.

2. Replace hand-rolled 'ManagedRuntime.make(defaultLayer, { memoMap })' with
   the existing 'makeRuntime(Service, defaultLayer)' helper from
   src/effect/run-service.ts. Same pattern as bus, sync, instance-store, etc.

3. Use 'Effect.cachedInvalidateWithTTL(populate, Duration.infinity)' for
   single-flight cache + manual invalidation. Concurrent first-callers now
   share one populate; refresh() invalidates so the next get() re-populates.

4. Restore eager initial refresh — Schedule.fixed waits the duration before
   the first run, which would have lost the original 'void refresh()' on
   module load. Now: refresh().pipe(Effect.andThen(refresh().repeat(...))).

5. Simplify fresh() mtime check — fs.stat's info.mtime is always Option<Date>;
   dropped the dead Option.isOption guard.

6. Drop unnecessary Effect.fnUntraced on loadFromDisk/loadSnapshot (these
   are values, not traced effects). Replaced loadSnapshot's manual try/catch
   wrapper with Effect.tryPromise + Effect.catch.

7. Effect.orDie on populate — Service interface declares get returns
   Effect<Record<string, Provider>> (E = never); failures from HttpClient,
   FileSystem, JSON.parse must die rather than silently leak.

Smoke + provider tests pass.
@kitlangton kitlangton force-pushed the kit/models-dev-service branch from 098a921 to 3a8a274 Compare May 2, 2026 16:34
The new ModelsDev Service wasn't included in createRoutes layer mergeAll, so
HttpApi handlers calling ModelsDev.Service.use() got an unhandled defect at
runtime, surfacing as a bare 500.
function url() {
return Flag.OPENCODE_MODELS_URL || "https://models.dev"
export interface Interface {
readonly get: () => Effect.Effect<Record<string, Provider>>
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a thunk?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADAM! Great question. Effect.fn returns a function always, and we're using that for free log spans and some stack tracing biz. Yeah, a bit diff from ZIO patterns.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effects are still lazy and it doesn't have to be a thunk to work, but Effect.fn is actually pretty cool.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aight, ty ty

kitlangton added 3 commits May 2, 2026 12:58
Mocks HttpClient via HttpClient.makeWith and pre-populates the on-disk cache
to drive every code path: disk-hit, disk-empty fallback, single-flight
dedup, in-memory cache stickiness, refresh fetch, fresh/stale TTL skip,
and HTTP-error swallow.

Layer.fresh is required to defeat the process-global MemoMap so each test
gets its own cachedInvalidateWithTTL state.
- Use Schedule.spaced (not fixed) so the periodic refresh waits between
  completions instead of firing twice on startup.
- Pull JSON.parse out of the Flock-held scope — keeps the cross-process
  lock release-eager.
- Only invalidate after a successful fetchAndWrite; the no-op fresh-skip
  and ignored-failure paths shouldn't force the next get() to re-read.
- Drop comments narrating obvious behavior; fix a misplaced WHY comment.
- Use HttpClient.make (matches mockHttpClient elsewhere) instead of
  makeWith with a Preprocess cast in the new tests.
Mutating Flag.OPENCODE_MODELS_PATH at module-load time leaked the change
into every subsequent test file in the same bun process — Provider state
init then read undefined, fell into the snapshot/fetch fallback, and on
slower CI lanes timed out or returned empty providers (surfacing as the
"no providers found" failure on Windows for the SDK no-reply parity test).

Wrap both mutations in beforeAll/afterAll so they only apply while this
suite runs.
@kitlangton kitlangton merged commit f8738c9 into dev May 2, 2026
10 checks passed
@kitlangton kitlangton deleted the kit/models-dev-service branch May 2, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants